AdaBoost and Support Vector Machines for Unbalanced Data Sets

Boost is a kind of method for improving the accuracy of a given learning algorithm by combining multiple weak learners to “boost” into a strong learner. The gist of AdaBoost is based on the assumption that even though a weak learner cannot do good for all classifications, each of them is good at some subsets of the given data with certain bias, so that by assembling many weak learner together, the overall accuracy is expected to be higher. Support Vector Machine (SVM) is a popular machine learning technique for solving classification and regression problems. In this project, LIBSVM tools of SVMs was used to solve classification problems. The AdaBoost.M1 algorithm utilized SVMs as component learners and the new algorithm was proved to boost the accuracy of unbalanced datasets sharply. In the best case, AdaBoost.M1 with SVM algorithm achieved accuracy improvement of 10%. However, AdaBoost was not always useful for performance boosting. In the worst case of the vowel dataset, the performance of AdaBoost.M1 with SVM was slightly worse than the grid search method. By exploring various aspects of AdaBoost.M1 with SVM algorithm, I found that the gamma update settings had an important impact on the accuracy. It effected the number of component learners as well as the generalization of each learner. Ideally, proper number of weak learners would fit in unbalanced training data very well.